An empirical study of the correlation between code coverage and reliability estimation

Existing time-domain models for software reliability often result in an overestimation of such reliability because they do not take the nature of testing techniques into account. Since every testing technique has a limit to its ability to reveal faults in a given system , as a technique approaches its saturation region fewer faults are discovered and reliability growth phenomena are predicted from the models. When the software is turned over to eld operation, signiicant overestimates of reliability are observed. In this paper, we present a technique to solve this problem by addressing both time and coverage measures for the prediction of software failures. Our technique uses coverage information collected during testing to extract only eeective data from a given operational proole. Execution time between test cases which neither increase coverage nor cause a failure is reduced by a parameterized factor. Experiments using this technique were conducted on a program created in a simulation environment with simulated faults and on an industrial automatic ight control project which contained several natural faults. Results from both experiments indicate that overesti-mation of reliability is reduced signiicantly using our technique. This new approach not only helps reliability growth models make more accurate predictions, but also reveals the eeciency of a testing proole so that more eeective testing techniques can be conducted.